Degrees of Impact: Analyzing the Effects of Progressive Librarian Course Collaborations on Student Performance

The Claremont Colleges Library conducted direct rubric assessment of Pitzer College First-Year Seminar research papers to analyze the impact of diverse levels of librarian course collaborations on information literacy (IL) performance in student writing. Findings indicate that progressive degrees of librarian engagement in IL-related course instruction and/or syllabus and assignment design had an increasingly positive impact on student performance. A secondary indirect analysis of librarian teaching evaluations and self-perceived learning gains by students and faculty showed no correlation to rubric IL scores, suggesting the importance of “authentic” assessment in determining actual learning outcomes. This mixed-methods study presents findings in each area and examines their implications for effective IL course collaborations. Introduction: Holistic Assessment and its Challenges Modern libraries operate in a climate of rapid organizational, technological, and information change, the demands of which are exacerbated by persistent resource scarcity. In this context, assessment has become central to the practice of determining and communicating the “value” of academic libraries to the communities of higher learning in which they are situated.i From ethnography to quantitative study to user experience research, numerous strategies are employed to evaluate the effectiveness and relevance of library services, tools, operations, and personnel in order to support the goal of producing “evidence-based reports of measurable impact.” ii Within this evaluation-focused framework, the established but ever-expanding pedagogical role of the librarian in higher education necessitates reliable methods for measuring not only the Degrees of Impact Booth, Lowe, Tagge, & Stone (C&RL, 2014) 1 of 43 CO LL EG E & RE SE AR CH L IB RA RI ES P RE -P RI NT teaching performance of library educators, but the actual learning effects of their interventions among students.iii Learner-centered IL evaluation follows global trends in higher education assessment that supplement quantitative analytics with “holistic” evaluation techniques involving authentic student work, and learning experiences that involve applied or “real world” elements beyond testing measures based on recall and reasoning.iv Connecting student academic performance to library/ian pedagogical and course collaboration efforts such as instruction and assignment design in these ways not only cements our role in facilitating the development of a diverse range of concepts and competencies that comprise information literacy (IL), it responds to a broader call for increased accountability in higher education.v Weiner argues that there is a “history of difficulty in integrating information literacy with the postsecondary educational process.”vi Similarly and by extension, macro and micro environmental factors in higher education present obstacles to holistic library learning assessment. For the librarian working in the context of non-credit bearing course-integrated or course-related instruction, common impediments include teaching scenarios of limited duration (e.g., the one-shot), lack of access to student coursework, little influence over course and assignment design, minimal faculty-librarian collaboration, and differing syllabi and assignment expectations across a unified program. At the program level, non-requisite credit-bearing IL courses, detachment from institutional governance and accreditation review, and/or frequent turnover among allied academic stakeholders are common challenges. Within one of the most typical programmatic IL instruction scenarios, shared first-year experience courses such as introductory seminars and foundational writing/rhetoric programs, these macro and micro challenges often meet. Varied degrees of course collaboration (e.g., Degrees of Impact Booth, Lowe, Tagge, & Stone (C&RL, 2014) 2 of 43 CO LL EG E & RE SE AR CH L IB RA RI ES P RE -P RI NT buy-in) can occur between individual teaching faculty and librarians, resulting in different levels of instruction and input into assignment design across a program. In the event of administrative buy-in and standardized IL outcomes, diverse teaching strategies and curricula are often still employed by different librarians within a given program based on faculty collaboration level and personal pedagogy. If it is assumed that direct, program-level assessment of student learning is dependent on the evaluation of shared outcomes or interventions, reliable and coordinated evaluation of student learning becomes problematic within these common constraints.vii Developing practical IL assessment frameworks that focus on authentic student output while also allowing for individualized librarian pedagogy and unique course collaboration scenarios will be critical for communicating the value proposition of libraries and their quantitative and qualitative contributions to student learning on a larger scale; it is within this context that the present study is situated. Research Motivation and Context The Claremont Colleges in Claremont, California are a consortium of seven contiguous but independent institutions situated around a common Library. The Claremont campuses comprise a total enrollment of roughly 7,000 students across five liberal arts colleges (Claremont McKenna College, Harvey Mudd College, Pitzer College, Pomona College, and Scripps College) and two graduate universities (Claremont Graduate University and Keck Graduate Institute). This consortial scenario presents unique challenges and opportunities for Claremont Colleges Library (CCL) staff involved in IL education, ranging from the need for customized pedagogies that reflect campus cultures to the ability to test creative approaches to resource scalability among different stakeholder groups. Degrees of Impact Booth, Lowe, Tagge, & Stone (C&RL, 2014) 3 of 43 CO LL EG E & RE SE AR CH L IB RA RI ES P RE -P RI NT For the last two decades, teaching librarians at CCL have provided instruction to first-year seminar programs at each of the undergraduate colleges to fluctuating degrees of depth – some programs received minimal opt-in coverage, while others featured a requisite course collaboration with an assigned librarian based on subject area ‘fit’. Since the establishment of its Instruction Services department in 2011, CCL has successfully augmented efforts to standardize course interventions and programmatically integrate outcomes-oriented IL instruction into the first-year seminars at all five of its undergraduate colleges. Steps toward greater “institutionalization”viii have been achieved through increasingly close collaboration with seminar program coordinators and campus assessment officers, faculty and librarian professional development initiatives, and the Library-led creation of a shared Colleges IL definition and firstyear seminar learning outcomes.ix Expansion of programmatic collaboration from two to five campuses has effectively doubled the amount of first-year instruction conducted by CCL teaching librarians, translating to over one hundred unique course collaborations that reach the vast majority of Claremont undergraduates at a foundational point in their college experience. These courses are distributed among approximately twenty teaching librarians, each collaboration featuring a unique syllabus with minimal topical overlap and an almost total lack of common assignments with the exception of a relatively standard final research paper. Evaluating student learning across these collaborations requires the application of assessment strategies that apply to highly individualized instruction scenarios that inevitably result in what we will describe as “progressive” degrees of IL interaction with faculty and students. Degrees of Impact Booth, Lowe, Tagge, & Stone (C&RL, 2014) 4 of 43 CO LL EG E & RE SE AR CH L IB RA RI ES P RE -P RI NT At Claremont, despite considerable coordination among teaching librarians and seminar coordinators, student-librarian and faculty-librarian interactions in the first year range from negligible to substantial across courses in the same program. This scenario creates unequal levels of student exposure to IL concepts, and, by extension, different learning effects based on the depth of faculty/librarian collaboration. Given this phenomenon, it is essential to examine whether relative degrees of faculty-librarian collaboration and/or librarian instructional interactions result in qualitative and/or quantitative changes in student learning. In the present study, investigators applied a mixed-methods analysis to student culminating research papers and library instruction evaluations produced in 13 course collaborations in Pitzer College’s First-Year Seminar (FYS) program with the goal of determining the student learning effects of differential levels of IL course engagement. Based on the near-universal FYS deliverable of a final research paper with a secondary source integration component, researchers determined that assessment of papers using an established, CCL-developed IL rubric (Appendix A) was the ideal method for conducting authentic evaluation of progressive IL interventions in student work. In order to examine whether perceived effectiveness of library instruction in terms of instructor performance and perceived student learning self-perceptions correlated to student IL performance as established by rubric analysis, researchers conducted a secondary analysis of web-based faculty and student survey evaluations of library instruction across the same course pairings. Rubric evaluation of student work indicates that increased librarian intervention in the form of more intensive IL instruction and/or assignment design collaboration with faculty had a marked and statistically significant positive impact on first-year student IL performance in researchDegrees of Impact Booth, Lowe, Tagge, & Stone (C&RL, 2014) 5 of 43 CO LL EG E & RE SE AR CH L IB RA RI ES P RE -P RI NT based writing. Secondary analysis of student/faculty evaluations of FYS librarian instruction revealed no correlation between IL performance in research papers and self-perceived student learning gains and/or perceptions of librarian teaching effectiveness. This dichotomy underscores the importance of direct, student-focused IL assessment in order to determine actual librarian intervention effects. This paper presents findings of each analysis (rubric and survey) and discusses their implications for constructing effective individual and programmatic course collaboration frameworks at the first year and beyond. Literature Review Assessment of information literacy instruction has a long-established presence in library education literature. Following broad trends within higher education assessment, a practice once focused on librarian performance evaluation in instructional contexts has shifted toward to a more direct and student-focused approach to learning assessment using a variety of techniques, including fixed-choice tests, performance assessments, and rubrics.x This holistic assessment trend is in large part a response to acknowledged limitations of test-based IL assessment; Dunn notes that “librarians have attempted to assess student information competence skills by ‘testing’ students with standard classroom tests based on multiple choice, fill-in-the-blank, and matching questions. Such tests . . . cannot assess the effectiveness of student search skills in real life situations.”xi Holistic, authentic, and mixed-methods assessment of library instruction is as a result becoming well-traversed territory.xii Among many emerging techniques in the field, rubric evaluation of student work is an increasingly used assessment strategy in libraries and higher education that seeks to address the limitations of test-based and instructor-focused performance evaluation. Degrees of Impact Booth, Lowe, Tagge, & Stone (C&RL, 2014) 6 of 43 CO LL EG E & RE SE AR CH L IB RA RI ES P RE -P RI NT Rubric evaluation is founded on the principle of outcomes-based and “student-centered” instruction, and Oakleaf in particular has made the case that rubric assessment is superior to test-based evaluation of student work.xiii Similarly, Daniels and Choinski, Mark, and Murphy’s studies demonstrate the effectiveness of rubric evaluation of student learning in an IL instruction context.xiv Previous research has shown positive effects of minimal or ‘one shot’ instruction on student IL concept retention and skill performance; Spievak and Hayes-Bonahan conducted a rigorous study that determined significant increases in student IL following exposure to a single IL instruction session.xv Analyses of the effects of curriculum-integrated IL instruction tend to find that intensive course collaborations result in improved student learning and performance; Gilbert and Gilbert determined that political science students who participated in a three-month lab course demonstrated increased IL skill performance over students that did not participate in the lab.xvi Hearn correspondingly demonstrated that a 10-session integrated IL model resulted in the integration of sources of higher quality as determined by citation analysis.xvii Positive learning outcomes due to IL instruction are not universal, however; Emmons and Martin found little change in student source use in a rubric-based preand postanalysis of papers before and after a program of library instruction.xviii Many authors have examined the efficacy of IL instruction within first-year seminar programs, while increased collaboration between librarians and subject faculty has been shown to have significant impact on the quality and breadth of IL instruction.xix Of particular relevance to the current project is Ghandi’s 2005 study in which a five-session, highly collaborative IL instruction model was compared to a one-shot IL instruction model in the same course; the former group of Degrees of Impact Booth, Lowe, Tagge, & Stone (C&RL, 2014) 7 of 43 CO LL EG E & RE SE AR CH L IB RA RI ES P RE -P RI NT students showed marked improvement in IL concept understanding relative to the one-shot group, as well as greater motivation and satisfaction with the IL course component.xx Significantly, in the five-session model the librarian was granted access to the course syllabus and tailored their instructional delivery to assignment needs, while both faculty and librarian emphasized IL within content delivery. The effectiveness of rubric evaluation, outcomes-focused instruction, collaboration with academic faculty/staff, and the impact of IL scaffolding throughout into the curriculum are wellestablished concepts in the literature. However, with the exception of Ghandi who examined the positive learning and motivation outcomes of a collaborative and course-integrated model compared to a one-shot session, there is a dearth of research that investigates the progressive effects of IL instruction in librarian-faculty curricular collaborations of varying intensity levels. xxi In addition, while educational researchers have demonstrated low levels of accuracy in student skills self-assessments (particularly among novices), few studies have compared self-perceived student IL learning and/or evaluations of librarian teaching effectiveness to ‘authentic’ student post-instruction performance; instead, most extant literature has instead investigated student skills self-evaluations compared to actual IL skills performance.xxii The present study seeks to address these gaps in the literature from a mixed-methods standpoint in order to 1) gauge the learning effects of IL course collaborations of varying depths and 2) validate the rigor of direct rubric assessment relative to student self-perceptions and instructional effectiveness evaluations.